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This report presents an analysis of the 
University’ s evaluation of its Academic 
Preparation and Educational Partnership 
Programs. Three recommendations are made 
for strengthening the evaluation processes in 
the future. 

The Commission concludes that the 
information reported offers reasonable 
evidence of program effectiveness, and 
therefore recommends that funding for these 
programs be restored in the 2006-07 State 
budget. 

The Commission also recommends that 
legislative funding decisions in subsequent 
years be informed by the extent to which the 
University continues to work cooperatively 
with the Commission in strengthening the 
evaluation process, as discussed in this report. 



Contents 

Background / 

Data and Research Issues Regarding the 

Accountability Framework 2 

Policy, Program, and Evaluative 

Recommendations 3 

Commission Summary of Findings by Program 
Area 3 



Commission Analysis of the University’s Findings. 4 
Conclusion and Recommendation for Funding 6 

The Commission advises the Governor and Legisla- 
ture on higher education policy and fiscal issues. 
Its primary focus is to ensure that the state’s edu- 
cational resources are used effectively to provide 
Californians with postsecondary education oppor- 
tunities. More information about the Commission 
is available at www.cpec.ca.gov . 



Background 

An important on-going educational concern of pub- 
lic officials across the nation is the extent to which 
public investments in academic preparation pro- 
grams are having desired effects on student aca- 
demic achievement and college preparedness. In 
other words, is the social and economic return on 
investment substantial enough to warrant continued 
funding of such programs? To address this ques- 
tion, the 2005-06 California Budget Act contained 
language requesting the University of California to 
provide a comprehensive report on student aca- 
demic preparation and educational partnership pro- 
grams. The language read: 

It is the intent of the Legislature that the uni- 
versity report on the use of state and univer- 
sity funds provided for these programs, in- 
cluding detailed information on the outcomes 
and effectiveness of academic preparation 
programs consistent with the accountability 
framework developed by the University in 
April 2005. The report should be submitted to 
the fiscal committee of each house of the leg- 
islature by no later than April 1, 2006. 

Pursuant to the above legislative language, the Uni- 
versity formed an Accountability Planning and 
Oversight Committee charged with the responsibil- 
ity of developing a new accountability framework. 
As shown in Appendix A, the Oversight Committee 
included legislative staff and representatives of 
various state agencies. 
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The overall mission of the University’s Student Academic Preparation and Educational Partnership 
(SAPEP) programs, as outlined in the 2005 Accountability Framework, is to: 

Work in partnership with K-12, the business sector, community organizations and other institu- 
tions of higher education to raise student achievement levels generally and to close achievement 
gaps between groups of students throughout the K-20 pipeline so that a higher proportion of 
California’ s young people, including those who are first generation, socioeconomically disad- 
vantaged and English language learners, are prepared for postsecondary education, pursue 
graduate and professional school opportunities, and/or achieve success in the workplace. 

The framework is intended to guide the delivery and evaluation of the University’s programs and estab- 
lishes three-year and five-year performance goals. 

The University currently administers 16 key academic preparation and partnership programs through 
four alliance themes: (1) K-12 Student Academic Preparation, (2) Community College Programs, (3) K- 
20 Partnership Programs, and (4) Graduate and Professional School Programs. The University reports 
that despite continued budget uncertainties, its sponsored programs are able to reach more than 116,000 
participants, including K-12 students, community college students, university undergraduate students, 
parents of K-12 students, and teachers, counselors, and school administrators. This figure excludes 
graduate and professional students and users of ASSIST, the on-line community college articulation sys- 
tem. 

The stakes associated with the 2005 Accountability Framework are high because funding for preparation 
programs has sharply declined, with combined State and University funds dropping from nearly $85 
mi llion in the year 2000-01 to $29 million in the 2004-05 year. The decline in funding is due partly to 
the effects of the 2002 economic recession and partly to the uncertainty regarding program effective- 
ness. Appendix B shows funding expended for academic preparation programs from 1997-98 through 
2005-06. Appendix C shows the number of individuals served and the average cost per student. At the 
moment, the Governor’s 2006-07 revised budget does not contain any funding for university preparation 
programs. It is evident that any future funding rests on a favorable review of the University’s report by 
the Administration and the Legislature. 



Data and Research Issues Regarding the Accountability Framework 

The 2005-06 budget language permits the university to use a range of data and evaluative protocols to 
assess program benefit and success. Accordingly, the UC report contains various types of data intended 
to inform program effectiveness. Essentially, these data can be organized under four broad research pro- 
tocols: (1) data describing the outcome characteristics of program participants (descriptive protocol); 
(2) participant outcome data that is compared with non-participant data (non-experimental protocol); (3) 
participant outcome data that is matched with meaningful comparison groups while controlling for other 
relevant socioeconomic and school performance factors (quasi-experimental protocol); and (4) student 
personal testimonies (qualitative/focus-group protocol). 

Although quasi-experimental protocols normally result in far greater insights into cause-effect relation- 
ships, the other three data protocols also yield useful information describing general outcome character- 
istics of program participants, while not explicitly asserting a cause-effect relationship. In other words, 
descriptive protocols can describe important outcome differences between program participants and 
non-participants, but they cannot address how much of the difference can be attributed to a particular 
academic preparation program and how much of the difference is a result of other factors. 

The California Postsecondary Education Commission has issued a number of evaluative reports over the 
past 25 years to assist the Governor and the Legislature in interpreting the results of academic prepara- 
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tion studies so that informed funding decisions could be made. This report contains an examination of 
the findings and research studies included or referenced in the University’s 2006 academic preparation 
program report. Significant attention was placed on design methodology and the validity of conclusions 
reached by the University regarding program effectiveness. Findings reached by the Commission were 
derived in part by comparing the performance goals outlined in the framework with the performance 
outcomes reported by the University. 

Because the framework is considered a work-in-progress, the Commission, in the next section, offers a 
set of programmatic and evaluative suggestions that would strengthen and improve future evaluations. 
More specifically, the Commission believes that the road ahead should be marked by (1) greater empha- 
sis placed on quasi-experimental research designs, (2) the development of a comprehensive longitudinal 
database that tracks program participants from secondary school to workforce careers, and (3) an effort 
to define more precisely key evaluative terms, such as return-on-investment and cost-effectiveness. 

Policy, Program, and Evaluative Recommendations 

1. For academic preparation programs that have scholastic selection criteria, it is recommended that 
the University strengthen its evaluation process by using quasi-experimental design methods to 
measure program effectiveness. At a minimum, this would mean that the outcomes of program 
participants be matched with the outcomes of a comparable control group, based in part on pro- 
gram selection criteria and relevant socioeconomic factors. Statistical methods, such as logistic 
regression and multivariate analysis of variance (ANOVA), should be used to separate out pro- 
gram effects from other external effect sources. Programs that have scholastic selection criteria 
include the Early Academic Outreach Program (EAOP), Mathematics, Engineering, Science 
Achievement (MESA), MESA Community College Transfer Program, School/University Partner- 
ship Program, and the Graduate and Professional School Academic Preparation Programs. 

2. It is recommended that the Commission analyst with primary responsibility for evaluating aca- 
demic preparation programs be appointed to the University’s Accountability Oversight Commit- 
tee to assist it in identifying appropriate research methods and data needs to enhance the program 
evaluation process. It is recommended that, to the extent possible, such efforts include develop- 
ment of a comprehensive longitudinal database capable of tracking student outcomes during and 
after program participation. Once a longitudinal database is developed, the marginal cost of 
evaluating individual programs will likely be reduced significantly. 

3. The Accountability Framework for UC academic preparation programs currently contains 
evaluation concepts such as cost-effectiveness and return on investment, whose meanings are not 
well understood. Therefore, it is recommended that the University work with the Commission in 
defining such concepts more precisely so that public officials have a clear understanding of the 
measures contained in subsequent evaluation reports and studies. 



Commission Summary of Findings by Program Area 

K- 1 2 Student Academic Preparation Programs 

Based on a comparison of performance goals with outcome data provided by the University, it appears 
that all programs in this category are on target for meeting three-year and five-year performance goals. 
As noted previously, the Commission recommends that programs with scholastic selection criteria, such 
as EAOP and MESA, be evaluated using quasi-experimental designs in order to validate program effec- 
tiveness. 
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UC Community College Programs 

A successful community college transfer function continues to be a higher education policy objective of 
the State. The University intends to establish transfer goals once 2005-06 transfer data, which will serve 
as a baseline, have been analyzed completely. Significant numbers of MESA and Puente program par- 
ticipants continue to transfer to baccalaureate institutions. During 2004-05, 100 percent of the 554 
MESA transfer students declared majors in math and science-based fields, consistent with the program’s 
primary performance objective. However, it is unclear how the University intends to measure improve- 
ments in transfer readiness. 

K-20 Partnership Programs 

Many social factors impact student success, including school environment, quality and content of in- 
struction, family support, and neighborhood safety. Given this realization, K-20 alliances were estab- 
lished with the aim of “establishing ties between campuses, schools, and local community and business 
organizations to promote consensus and joint action to raise student achievement levels and to address 
the barriers to educational equity.” 

Data reported by the University indicate that the ArtsBridge Program, the Presuss Charter School, and 
the programs comprising K-20 Regional Intersegmental Alliances as a whole are meeting or exceeding 
performance targets. Outcomes data were not reported separately for each individual intersegmental al- 
liance program, so it is not possible to comment on their individual effectiveness. The University in- 
tends to establish performance goals related to college readiness and college-going for its partnerships 
with selected schools and faith-based organizations once 2004-05 baseline data has been analyzed. 

K-20 Graduate and Professional School Programs 

The University’s graduate and professional programs seek to identify highly qualified prospective 
graduate students from economically and educationally disadvantaged backgrounds and prepare them 
for careers as researchers, academics, specialists, and leaders. Performance data reported indicate that 
about 78% of undergraduate students that participate in UC graduate and professional school prepara- 
tory programs eventually enroll in graduate or professional programs. 



Commission Analysis of the University’s Findings 

Analysis of K- 1 2 Student Academic Preparation Programs 

Programs included in this category consist of: (1) the Early Academic Preparation Program (EAOP); (2) 
the Mathematics and Science Achievement Program (MESA); (3) the Puente Project High School Pro- 
gram; (4) Student Initiated Programs; (5) UC College Prep Online; and (6) UC Links. The data pro- 
vided by the University indicates that all of these programs are on target for achieving 2010 target goals 
for college readiness and college going rates. The data also indicate that the performance rates of pro- 
gram participants are generally higher than overall statewide rates. For example, it is estimated that 
73.7% of EAOP students and 54.2% of sampled MESA students completed the university’s A-G college 
preparatory curriculum, compared with a statewide rate of 33.7%. 

Many of the programs in this category have scholastic selection criteria, which means that it is invalid to 
infer program effectiveness by comparing the success of these program participations to statewide rates. 
What is needed are comparable or matched control groups. One study (Quigley 2003), which was cited 
in the report as offering clear evidence of the effectiveness of EAOP, used appropriate comparison 
groups, while controlling for other potential effects. The conclusion reached by the program researcher 
is that EAOP students are more than twice as likely as non-EAOP students to complete the A-G re- 
quirements. 
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The Commission reviewed the Quigley study and found several concerns. First, the researcher asserted 
that the effect of EAOP could be best understood by examining an odds ratio interaction effect of stu- 
dent ability (high ability vs. low ability) by school type (EAOP Schools vs. Non-EAOP Schools). Al- 
though the design was valid, the researcher did not report the college preparatory rates for appropriate 
comparison groups, so it is not clear how high ability students attending EAOP schools compare to high 
ability students attending similar schools that do not offer EAOP (see Display 1). In the absence of such 
comparisons, it is difficult to draw a definitive conclusion from this study regarding the effectiveness of 
EAOP. Second, a major finding not pointed out the reports conclusion was that students who completed 
9 th grade English with at least a B grade were more than 12 times more likely to complete the A-G cur- 
riculum than those students who did not do so. This later effect is substantially larger than the calcu- 
lated EAOP effect and should have been discussed in the report. Although the conclusion of the Quig- 
ley study is incomplete, it was a well-designed study that used appropriate quasi-experimental methods, 
and therefore, could serve as model for assessing program effectiveness when comparable control 
groups are required. 

Display 1 Illustration of Data not Included in the a Study by Quigley (2003) Regarding the 
Effectiveness of the University’s EAOP Program 





High Ability Students 


Low Ability Students 


Students Attending EAOP 
Schools 


.40 


? 


Students Attending Schools 
Not Offering EAOP 


? 


? 



Note : Cell data indicate the proportion of students that completed UC’s A-G requirements. High-ability 
refers to students with a scholastic record required for EAOP participation. 



Analysis of UC Community College Programs 

Programs in this category consist of: (1) UC community college transfer programs, (2) the MESA Com- 
munity College Program, (3) the Puente Community College Program, and (4) the UC Community Col- 
lege Articulation Program called ASSIST. The University’s accountability framework will be updated 
to include numerical transfer goals to enhance the transfer function once a baseline has been established 
using 2005-06 transfer data. No agreements have been reached to date on the type of data and informa- 
tion needed to reliably measure improvements in transfer readiness across majors. 

Performance goals related to ASSIST include increase in use of the system and continued improvement 
in usability. The University reports that over 700,000 people used ASSIST in the 2004-05 academic 
year, up from 600,000 visitors in 2003-04. ASSIST users viewed over 6.7 million course articulation 
reports, up from 5.2 million in 2003-04. Virtually all of the majors at all UC campuses are now articu- 
lated with every community college in the state. Approximately 75% of the responses to a recent survey 
were considered by the University to be very positive. The negative responses were related to a desire 
for information beyond of the scope of ASSIST, such as articulation with independent and out-of-state 
colleges and universities. 

Evaluation data indicate that the University’s MESA and Puente Community College Program are ex- 
periencing a high degree of success in improving the participation of underrepresented students in four- 
year institutions. Puente community college transfers to four-year institutions grew by 78% between 
1999 and 2004, from 295 to 526 students. This program provides an academically rigorous two-year 
English sequence, combined with intensive counseling and UC mentors. During 2004-05, 554 MESA 



Page 5 








California Postsecondary Education Commission 



students transferred to baccalaureate institutions, with 100 percent of these students declaring majors in 
math or science fields. What is not known, however, is the proportion of MESA students for a given 
community college cohort that were transfer ready within a two or three-year period as a result of 
MESA participation. Such a statistic, when compared with a control group comprising prospective math 
and science majors with a demographic and scholastic makeup similar to that of MESA students, would 
provide a more revealing indication of program impact and effectiveness. 

Analysis of K-20 Partnership Programs 

Programs in this category consist of: (1) the ArtsB ridge Literacy/Language Program, (2) Community 
Partnerships, (3) the Presuss Charter School at UC San Diego, and (4) K-20 Regional Intersegmental 
Alliances. Performance goals are related to literacy/language skills, successful completion of the UC 
visual/performing arts requirement, college readiness and enrollment rates, service learning for teachers, 
and leveraging State funding investments with private and corporate endowments. Data provided by the 
University indicate that the ArtsBridge Program, the Presuss Charter School, and the K-20 Regional In- 
tersegmental Alliances as a whole are meeting or exceeding performance targets. 

For example, it is reported that 64% of ArtsBridge participants improved at least 20% on pre/post vo- 
cabulary assessment and that 45% of participants scored 70% or higher on the post project vocabulary 
assessment. As another example, the Presuss Charter School, which serves low-income students, re- 
ceived the second highest API score (848) in its district. No performance data was recorded for UC’s 
community partnerships with schools and community and faith-based organizations. The university in- 
tends to use 2005-06 data as baseline information and then develop appropriate numerical goals pertain- 
ing to college going and college readiness. 

One concern of the Commission is that the University did not explicitly report outcomes for each indi- 
vidual K-20 Regional Intersegmental Alliance. The system elected instead to highlight selected success- 
ful sights, such as the Santa Ana Partnership with UC Irvine, the UC Berkeley Partnership, and the UC 
Santa Cruz Alliance. 

Analysis of Graduate and Professional School Programs 

Performance data reveal that about 78% of participants enroll in graduate school within three to ten 
years after participating in a graduate academic preparation program. The Commission believes that an 
additional measure of program impact could be obtained if the University disaggregated the data by 
gender and ethnicity and reported enrollment data for medical program participants separately. 



Conclusion and Recommendation for Funding 

The Commission concludes that the information reported by the University offers “reasonable” evidence 
of program effectiveness, and therefore, recommends that funding for these programs be restored in the 
2006-07 State Budget. The Commission further recommends that in order for these programs to con- 
tinue to receive funding that a more comprehensive quantitative evaluation take place for each program 
to access the value added by that program. If the Administration concludes otherwise, the Commission 
recommends that such findings and the rationale be made known to the University and to the general 
public. 

The Commission also recommends that legislative funding decisions in subsequent years be informed by 
the extent to which the University continues to work cooperatively with the Commission in strengthen- 
ing the evaluation process, as discussed in this report. 
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Appendix A 

Student Academic Preparation and Educational Partnerships 
Accountability Planning and Oversight Committee 

Committee Chairperson: Harold Levine, L/C Davis School of Education 



Marvin Alkin 


UCLA School of Education 


Danny Alvarez 


Senate Budget and Fisccd Review CommitU 


Michael Brown 


UC Santa Barbara/Board of Admissions 
and Relations with Schools (BOARS) 


Kathleen Chavira 


Senate Education Committee 


Catherine Cooper 


UC Santa Cruz/UC Office of the President 


Winston Doby 


UC Office of the President 


Neal Finkel stein 


WestEd 


Ron Fox 


California Department of Education 


Patricia Gandara 


UC Davis School of Education 


Marlene Garcia 


Senate Office of Research 


Elizabeth Halimah 


UC Berkeley 


Bruce Hamlett 


Assembly Higher Education Committee 


Celia Mata 


Assembly Budget Committee 


Hugh Mehan 


UC San Diego 


Debora Obley 


UC Office of the President 


Jeannie Oropeza 


Department of Finance 


Lynn Podesto 


Department of Finance 


Oscar Porter 


UC Office of the President 


Patrick Shields 


SRI International 


Anthony Simbol 


Legislative Analyst’s Office 


Jack Sutton 


UCLA 


Sara Swan 


Department of Finance 
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University of California 

Student Academic Preparation and Educational Partnerships 
1997-98, 2000-01, 2004-05, and 2005-06 Budgets 
($000s) 





1997-98 
State & 
UC Funds 


2000-01 
State & 
UC Funds 


2004-05 
State & 
UC Funds 


2005-06 
State & 
UC Funds 


Direct Instructional Programs 


Preuss Charter School 


$ 


$ 1,000 


$ 1,000 


$ 1,000 


UC College Preparation (online courses) 


- 


8,400 


3,106 


3,106 


Statewide Infrastructure Programs 


ASSIST 


360 


360 


429 


429 


Community College Articulation 


- 


- 


600 


600 


Direct Student Services Programs 


Community College Transfer Programs 


1,718 


5,295 


1,279 


1,279 


EAOP 


4,794 


16,094 


8,914 


8,914 


Graduate and Professional School Programs 


1,893 


8,575 


2,661 


2,661 


MESA K- 12 Programs 


4,169 


9,355 


4,861 


4,861 


MESA Community College Programs 


22 


1,309 


327 


327 


Puente High School 


- 


1,800 


1,051 


1,051 


Puente Community College Programs 


162 


757 


450 


450 


Student-Initiated Programs 


- 


- 


440 


440 


UC Links 


- 


1,656 


694 


694 


Community Partnerships 


- 


1,096 


312 


312 


ArtsBridge 

Longer-Term Strategies Originally Funded by the University 


1,500 


112 


112 


K-20 Regional Intersegmental Alliances 
(formerly School-LTniversity Partnerships) 


' 


15,591 


1,395 


1,395 


Other Programs 


Evaluation * 


- 


1,386 


820 


1,180 


UC ACCORD * 


- 


809 


360 


- 


Other Programs 

Programs that have been eliminated or consolidated into others, 
including Test Preparation, Dual Admissions, Gateways, 
Informational Outreach and Recruitment, Central Valley 


203 


1,291 


512 


512 


Programs 


4,750 


8,908 


- 


- 


Total 


$ 18.071 


$ 85,182 


$ 29.323 


$ 29.323 


General Funds 


[$ 16,996] 


[$82,243] 


[$ 17,323] 


[$ 17,323] 


University Funds 


[$ 1,075] 


[$ 2,939] 


[$ 12,000] 


[$ 12,000] 



*For 2005-06, funding previously budgeted for UC ACCORD is being used for evaluation. 
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Estimated Cost Per Studemt 
UC SAPEP programs compared to selected 
Federally funded academic preparation programs 



Program 


2005-06 State 
and UC Funds 


Average Number 
of Students 
Served 2003-04 
to 2004-05 


Average Cost 
Per Student 


K-12 Student Academic Preparation 








EAOP 


$ 8,914,000 


43,743 


$ 204 


MESA K-12 Programs 


4,861,000 


16,620 


292 


UC College Preparation (online courses) 


3,106,000 


4,164 


746 


Puente High School 


1,051,000 


3,876 


271 


Student-Initiated Programs 


440,000 


8,966 


49 


UC Links 


694,000 


3,960 


175 


Comparable federal programs: 








Upward Bound Classic 






4,721 


Talent Search 






377 


Community College Programs 








ASSIST 


429,000 


630,000 


1 


Community College Transfer Programs 


1,279,000 


18,488 


69 


MESA Community College Programs 


327,000 


3,559 


92 


Puente Community College Programs 


450,000 


9,719 


46 


K-20 Educational Partnerships* 








ArtsBridge 


112,000 


7,021 


16 


Community Partnerships 


312,000 


5,860 


53 


K-20 Regional Intersegmental Alliances 


1,395,000 


151,911 


9 


Preuss Charter School 


1,000,000 


766 


1,305 


Comparable federal programs: 








GEAR UP 






207 


Graduate and Professional School Programs 


2,661,000 


558 


4,773 


Comparable federal programs: 








McNair Postbaccalaureate Achievement Program 






10,185 



* includes all students at schools served 

Source of federal program costs per student: US Department of Education website 
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